Skip to content

fix(writer): a text column starts a new dictionary instead of giving up its dict layout (#650) - #679

Merged
dfa1 merged 1 commit into
mainfrom
fix/650-string-dict-layout
Nov 6, 2026
Merged

dfa1 merged 1 commit into
mainfrom
fix/650-string-dict-layout

Conversation

@dfa1

@dfa1 dfa1 commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Stacked on #677. Fixes the layout half of #650.

Rust's DictStrategy admits a text column by its first 8192 rows (not more than half distinct) and closes the open dictionary at 65,535 entries or 1 MiB, starting another. Java capped the column at 32,768 distinct values in its first batch and demoted it to per-chunk dictionaries beyond.

  • DictColumnState.ingestVarBinChunk ingests the prefix that fits Rust's constraints (first entry always admitted); the writer keeps a list of dictionaries per column and writes a chunked layout of dict layouts.
  • Admission looks at the first 8192 rows.
  • Interop test vs vortex-jni: same dict-layout count for pools of 100–150000 values, and for a sliding window that fills two dictionaries.

Not covered: nullable text still goes through masked(dict) per chunk when the layout is not admitted (the other half of the issue's bi-generico rows), and the OnPair/FSST choice inside the values.

🤖 Generated with Claude Code

https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV

…up its dict layout (#650)

Rust's dict layout admits a text column by its first 8192 rows (not more than half distinct) and,
when the open dictionary reaches 65,535 entries or 1 MiB, closes it and starts another
(DictStrategy, DictLayoutConstraints). Java capped the whole column at 32,768 distinct values in
its first batch and demoted it to per-chunk dictionaries beyond, so columns with locality (URLs,
titles, referers) lost the shared dictionaries and their bare bit-packed codes.

A text column now keeps a list of dictionaries, each filled to Rust's constraints (first entry
always admitted, the row needing an entry that no longer fits starts the next dictionary) and
written as a chunked layout of dict layouts. Admission looks at the first 8192 rows, as Rust does.
The interop test compares the number of dict layouts with vortex-jni for pools of 100 to 150000
values, and for a sliding window that fills two dictionaries.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
@dfa1
dfa1 force-pushed the fix/650-string-dict-layout branch from d699906 to c2895e3 Compare November 6, 2026 20:57
@dfa1
dfa1 merged commit a1df400 into main Nov 6, 2026
6 checks passed
@dfa1
dfa1 deleted the fix/650-string-dict-layout branch November 18, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant